Papers with Byte-pair encoding
Char2Subword: Extending the Subword Embedding Space Using Robust Character Compositionality (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Byte-pair encoding (BPE) is a ubiquitous algorithm in the tokenization process of language models but is only based on pre-training data statistics. |
| Approach: | They propose a character-based subword module that learns the subword embedding table in pre-trained language models like BERT. |
| Outcome: | The proposed method significantly improves the performance on the social media linguistic code-switching evaluation (LinCE) benchmark. |
BPE-knockout: Pruning Pre-existing BPE Tokenisers with Backwards-compatible Morphological Semi-supervision (2024.naacl-long)
Copied to clipboard
| Challenge: | Subword tokenisation is a mainstay in natural language processing (NLP) it allows the representation of an infinite space of text with a finite set of units. |
| Approach: | They propose to use byte-pair encoding to represent an infinite space of text with a finite set of units by removing subwords from the BPE vocabulary without impeding further use of merges that relied on them. |
| Outcome: | The proposed method improves token-based tasks in English, Dutch and German while saving training costs. |
Interpreting Topic Models in Byte-Pair Encoding Space (2025.coling-main)
Copied to clipboard
| Challenge: | Byte-pair encoding (BPE) is a popular method of tokenizing valid words onto a token space V b with a predetermined fixed size, and handling out-of-vocabulary words, breaking words into smaller tokens. |
| Approach: | They propose to interpret the recovery of valid words from these tokens as a ranking problem and apply existing evaluation measures to topic sets. |
| Outcome: | The proposed model interprets the recovery of valid words from these tokens as a ranking problem and applies existing evaluation measures. |